[ET Device Support] Define et_copy runtime h2d and d2h copy ops by Gasoonjia · Pull Request #18729 · pytorch/executorch

Gasoonjia · 2026-04-06T23:36:16Z

Stack from ghstack (oldest at bottom):

[ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline #18788
[ET Device Support] Extract shared device test utilities to reduce redundancy #18762
[ET Device Support] Add device tensor helper functions to TensorPtr API #18761
[ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies #18760
[ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries #18730
-> [ET Device Support] Define et_copy runtime h2d and d2h copy ops #18729
[ET Device Support] Define AOT device copy ops registry #18728
[ET Device Support] CudaAllocator: device memory allocator for CUDA backend #18477
[ET Device Support] Module: allocate device memory for planned buffers #18476
[ET Device Support] MemoryManager: add per-buffer device metadata #18475
[ET Device Support] MethodMeta: expose per-buffer device placement API #18474
[ET Device Support] DeviceMemoryBuffer RAII class for device memory lifetime management #18473
[ET Device Support] Emitter reads non_const_buffer_device from graph meta #18472
[ET Device Support] Device-aware memory planning: separate buffers per device type #18375
[ET Device Support] Add NonConstBufferDevice schema for per-buffer device mapping #18330
[ET Device Support] DeviceAllocator interface and DeviceAllocatorRegistry #17535
[ET Device Support] Annotate device attributes of CUDA backend IO tensors cuda device #18080
[ET Device Support] Parse device info from serialized tensor in tensor_parser #18328
[ET Device Support] Propagate device info from TensorSpec into serialized Tensor #18079
[executorch] Propagate device metadata from partitioner result onto TensorSpecs #18078

Implement C++ runtime kernels for device copy ops using DeviceAllocator:

h2d_copy_out: infers device from out tensor, calls
DeviceAllocator::copy_host_to_device
d2h_copy_out: infers device from self tensor, calls
DeviceAllocator::copy_device_to_host
Registered via EXECUTORCH_LIBRARY macro

Differential Revision: D99636776

Implement C++ runtime kernels for device copy ops using DeviceAllocator: - h2d_copy_out: infers device from out tensor, calls DeviceAllocator::copy_host_to_device - d2h_copy_out: infers device from self tensor, calls DeviceAllocator::copy_device_to_host - Registered via EXECUTORCH_LIBRARY macro Differential Revision: [D99636776](https://our.internmc.facebook.com/intern/diff/D99636776/) [ghstack-poisoned]

pytorch-bot · 2026-04-06T23:36:42Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18729

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 64 New Failures

As of commit e80c75a with merge base b5ae0b9 ():

NEW FAILURES - The following jobs have failed:

Build Presets / apple (pybind) / build (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2
Build Presets / linux (pybind, linux.2xlarge, executorch-ubuntu-22.04-clang12) / build (gh)
Build Presets / linux (pybind, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Build Presets / windows (pybind) / build (gh)
Process completed with exit code 1.
periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 2df372a50a898db2fc87675bdda2083fffad00187101a2d22f1ce91b2a9e2542 /exec failed with exit code 1
periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t d615c351e94e3b5b4d4f03b27ec93af7e0b39559058d5a24c95de05e705ae322 /exec failed with exit code 1
periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 1371ab22a3bb594a4e4ea93aa661b49486fc9b1dab2a6cba27838496596b910b /exec failed with exit code 1
pull / android / build-android (gh)
Process completed with exit code 2.
pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux-android / linux-job (gh)
RuntimeError: Command docker exec -t fffc01b1b5a2c1448c9b468c830aa3a636612cf2a51c2bdcdf0933c1b6c7df65 /exec failed with exit code 2
pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-lora-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-lora-multimethod-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-mcu-cortex-m-backend / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 45091bbb9aa968f54b28c3acf2d8d91c8a7a6727b37062e855cf5684508d7364 /exec failed with exit code 1
pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 070d3bfa1f6058f3c18ec5e4ac35ae46d966b3bdb0dfa40fa96ee685fe21c4b2 /exec failed with exit code 1
pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 02662a9540179292242cbde5d7fe3e6752e19a3aa6b1f6db8d336d33c6a50633 /exec failed with exit code 1
pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 3cacc18d06a770a6bd040a9b2c60aa86d218c9ef8dd2c4c3b8aa06f15d55ec0a /exec failed with exit code 1
pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 4adf93506491050c0b542668ba8e5244f7b7be7f1eb4b2e80d9f0bc5c4cdba7b /exec failed with exit code 1
pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t b0de953b6b5b7253cdc3f267a0fda66c3b2c222432e267fce43827b918a3e272 /exec failed with exit code 1
pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 33ef2c7f7ad59f81281577229ace17ab7c8608e864433826f3db28d7f6a3d376 /exec failed with exit code 1
pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 09af7745c68134558d9014caddf8c0ed8086a5363c29bd46533d373de32c6e17 /exec failed with exit code 1
pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t bcad491b0c0150c4c3cf9849b584d0bfc81affe8d6a997f5674ff161cc63b3ff /exec failed with exit code 1
pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t f485663f664f179ecebfd1d1138b32204e1d2bbb4f699a4a1b838d6ffe5a085c /exec failed with exit code 1
pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t f25f31626130b3ef32f0bf96e453e6b8f3a728fd1987618ea949b34bcdec77c0 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 7c8a041e731c485e50b1df2ff5834c04ddf91ef9c7228d1224e203c44a0b9e65 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 6d124540637e35e127cac5b24f70bd3b7d0410eb71db50ce760711f1a15c2615 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t ed59dbb931e71d62c02f32b09e441c8c0a951d44d184418ac2ab83e5998ce5f9 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t fbcafa43a87c6587b7f0f4539db4a9cb6b8c4da19b58f6c55b93a0f9faa91664 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 6f7974d4d369c98a7260abab1509281d209ff42694c9cccfa8f48b8e403cb969 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t 45579821518f62daa31d9f5d961ee53a55b26c749010ab79be90ac61b33cb724 /exec failed with exit code 1
pull / test-moshi-linux / linux-job (gh)
test_mimi
pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
RuntimeError: Command docker exec -t 546bf52c03cd1194e311728bef1ad30640d8498bf0b81359995bad881a413c90 /exec failed with exit code 1
pull / test-openvino-linux / linux-job (gh)
RuntimeError: Command docker exec -t ccac7a4b18454132dfa6990b4d6dd74b0aa1447313ec305d8bf48d54cd86f051 /exec failed with exit code 1
pull / test-parakeet-xnnpack-linux / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-phi-3-mini-runner-linux / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-vulkan-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t b2cea8ee7c248aa42853ffe161fea35865f628def4ab20ea6ab3bb4efef711cf /exec failed with exit code 1
pull / test-vulkan-operators-linux / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
pull / unittest / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest / macos / macos-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-arm-backend-with-no-deps (test_pytest_models_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t 25d68057ab35751d0caccea5a51ab231305f5e49ef8ef881c221bbef55e681fe /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_no_target) / linux-job (gh)
RuntimeError: Command docker exec -t 0ffc3ce77358e5939cdce946914b0c7853c540f55c77e602a271e8adf0ebc6ae /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t 5ce549bc39cec4bc0100aa0f1138a3c6978d54501335b4016e79cda7c5edff37 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / macos / macos-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-nxp-neutron / linux-job (gh)
RuntimeError: Command docker exec -t f0d05cdfb9c841b518ec003f718070ffc5bca3fb4b6b529dfb0283a5145a547b /exec failed with exit code 2
pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
RuntimeError: Command docker exec -t beeadfc484e936c7dce5cdb32e765c6b58b9cac41790bb4e15b0fe477ed953ba /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-04-06T23:37:10Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…y ops" Implement C++ runtime kernels for device copy ops using DeviceAllocator: - h2d_copy_out: infers device from out tensor, calls DeviceAllocator::copy_host_to_device - d2h_copy_out: infers device from self tensor, calls DeviceAllocator::copy_device_to_host - Registered via EXECUTORCH_LIBRARY macro Differential Revision: [D99636776](https://our.internmc.facebook.com/intern/diff/D99636776/) [ghstack-poisoned]

digantdesai

Review automatically exported from Phabricator review in Meta.

Gasoonjia requested a review from manuelcandales as a code owner April 6, 2026 23:36

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 6, 2026

Gasoonjia mentioned this pull request Apr 6, 2026

[ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries #18730

Open

meta-codesync bot added fb-exported meta-exported labels Apr 6, 2026

digantdesai approved these changes Apr 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ET Device Support] Define et_copy runtime h2d and d2h copy ops#18729

[ET Device Support] Define et_copy runtime h2d and d2h copy ops#18729
Gasoonjia wants to merge 2 commits intogh/gasoonjia/159/basefrom
gh/gasoonjia/159/head

Gasoonjia commented Apr 6, 2026 •

edited

Loading

Uh oh!

pytorch-bot bot commented Apr 6, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 6, 2026

Uh oh!

digantdesai left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Gasoonjia commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18729

❌ 64 New Failures

Uh oh!

github-actions bot commented Apr 6, 2026

This PR needs a release notes: label

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Gasoonjia commented Apr 6, 2026 •

edited

Loading

pytorch-bot bot commented Apr 6, 2026 •

edited

Loading

This PR needs a `release notes:` label